empug-0.4-local.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

<html>
<head>
  <meta name="generator" content="Bluefish 1.0.7">
  <title>Emping User Guide, Version 0.4</title>
</head>

<body bgcolor="white" text="black">
  <p align="center"></p>

  <center>
    <h1>Emping User Guide, Version 0.4</h1><font size=
    "4"><i>Author: Hans van Thiel, March 2008</i><br>
    email: hthiel.char@zonnet.nl</font>
  </center>
  
  <h1>New in Version 0.4</h1>
  <ul>
   <li>Emping 0.4 supports blank fields, which allows for merging of tables, 
   and thus a larger class of reduction problems.<br>
    This feature has, at the time of writing, only been summarily tested.</li> 
   <li>Emping 0.4 has a GUI, written in Gtk2Hs, which is the the Haskell port of GTK+</li> 
   <li>Source code and compilation have been optimized and GHC 6.8 has been used. Emping 0.4 
   appears to be faster than version 0.3</li>  
  </ul>

  <h1>1. Overview</h1>

  <h2>1.1. What</h2>

  <p>Emping is a utility that derives heuristic rules from nominal
  data. Nominal data are qualitative and unordered, as in:</p>

  <ul>
    <li>Color: red, green, blue, yellow, black</li>

    <li>Proposition 1: True, False</li>

    <li>Class: A,B,C,D</li>
  </ul>

  <p>Class is actually an ordinal attribute, but when the order is
  disregarded, it is nominal.</p>

  <p>Heuristic rules consist of attribute values (predicates) that
  together imply another attribute value. For example:</p>

  <dl>
    <dt>Color:green and Proposition 1:True is Class:B</dt>
  </dl>

  <p>Heuristic rules are purely empirical, with no foundation in a
  theory or model. The input of Emping is just a table of nominal
  facts. The user has to select which attribute is to be the
  consequent. Then Emping derives all shortest rules which, in the
  table, imply the values of the selected consequent. Each reduced
  rule is a generalization of one or more original rules, and
  therefore reduced rules may imply other reduced rules or be
  equivalant to others. If this is the case, these logical
  dependencies are also derived.</p>

  <h2>1.2. How</h2>

  <p>Emping reads a file in a comma seperated format (.csv) as
  produced by the Open Office Calc spreadsheet, and returns the
  results as .csv files that can be read by OO Calc.</p>

  <p>Emping 0.4 has a GUI and you can start it like any other application.</p>
  <p><img align="middle" src="Emping-1.png" alt="Emping-1" /> </p>  

  <p>The general idea is straightforward; see the example for details. First,
  you check or uncheck in the options menu whether you want to check the data for duplicates.
  These have no effect on the result, but will slow the program down, and they are automatically removed 
  if the option is selected (but not from the source file itself). If there are duplicates you
  can see which, with their frequencies, by saving them.</p>
  <p>Next you select the consequent attribute from a popup menu. If you have the relevant options menu item
  checked, emping will look for ambiguous rules for the selected consequent. These are
  rules which are the same, except for the consequent value. Emping works seamlessly with 
  ambiguous rules (but see the included white paper "Deriving
  Heuristic Rules from Facts" for how they effect the results.)</p>
  <p>Pressing the Reduce button will then start the reduction. Depending on the data set, this 
  may take a while, during which time the file save window will remain open. For small data sets
  the effect will not be noticeable, but do not try to close it unless you want to abort the reduction.</p>
  <p>Reduced rules may themselves imply, or be equivalent to, other reduced rules with the same consequent value.
  The default Top button will show only the top level of these rules (saved in a .csv file) but you can get all
  interdependencies through the Abduce All menu item.</p>
  <p>The reduced normal form file, the top level rule file and the
  others (if present), can now be loaded into OO Calc.</p>

  <h2>1.3. More</h2>

  <p>More about the principles on which emping is based can be
  found in the mentioned white paper "Deriving
  Heuristic Rules from Facts". The general idea is also illustrated by:
  <a href="http://j-van-thiel.speedlinq.nl/pdm/exq.html">Fishing</a></p>

  <h1>2. Example</h1>

  <h2>2.1. Step 1</h2>

  <p>Enter the data in Open Office Calc as shown:</p>

  <p><img align="middle" src="Mushrooms.png" border="0" alt=
  "Mushrooms in OO Calc"></p>

  <ul>
    <li>The first row must list the attribute names. It is recommended
    you give each a unique name, but you can use duplicates and blanks.
    The columns below each attribute name list the values for that attribute, possibly blanks. 
    <li>Each value name has its column as its scope, so you can use values "Yes" and "No",
    for example, for different attributes.</li>

    <li>A blank value field stands for "none of the others". This feature is new to version 0.4. 
    Using blanks you can now use Emping on data sets which have splits on some values. For example,
    an attribute "owns a car" can be 'yes" and 'no", and then "yes" can have fields on "price range", "make",
    and so on, with field "no" blank on those values. This feature has, at the time of writing,
    only be tested on small hypothetical data sets.</li>
    
    <li>You can select an attribute with blank fields as the consequent. Because a blank field
    stands for "none of the others" or "not applicable", rules with a blank consequent value
    will automatically be removed from the rule set before the reduction is applied.</li>

    <li>you can use whole numbers, like 0, 1, 3, 22 etc., but they
    will be treated as nominal values, just like A,B,C etc.</li>
  </ul>

  <h2>2.2. Step 2</h2>

  <p>Save the spreadsheet table in Text CSV format. Choose double quotes as the
  text delimiter (default). Whole numbers will be stored without
  delimiters, and emping will use them after checking if they are
  all digits (no negatives, no fractions). Names within quotes
  should not contain special characters, only letters, possibly
  numbers and white space.</p> 

  <h2>2.3. Step 3</h2>

  <p>Just double click on the emping file or type the program name in the command line. Then open the source
  file and apply the toolbar sequence.</p>  

  <h2>2.4. Step 4</h2>
  <p>This is what you see when the reduction is finished with "Class" as the consequent.</p>
  <p> There are 165 dependency trees, and 5 rules which are not implied by others 
  (except for possible equivalences, which will be shown in the .csv file).</p>
  <p><img align="middle" src="Emping-2.png" alt="Emping-2" /> </p> 
  <p>You can now choose to  save only the top level, all dependencies, or both.</p>
  <h2>2.5. Step 5</h2>

  <p>View the reduced normal form in a .csv file in OO Calc.</p>

  <p><img align="middle" src="Mushrooms_Class.png" border="0" alt="Mushrooms_Class" /></p>

  <p>View the top level reduced rules, including equivalences, in your saved file.
  Equivalences, shown as 'equals', are reduced rules which imply each other, both ways.</p>

  <p><img align="middle" src="Mushrooms_Class_Top.png" border="0" alt="Mushrooms_Class_Tops" /></p>

  <p>If you have chosen to save them, all logical implications are
  shown in that file.

  <p><img align="middle" src="Mushrooms_Class_All.png" border="0"  alt="Mushrooms_Class_All" ></p>

  <p>This file shows the partial order of all the reduced rules. Each inference chain, including any
  equivalences, is shown seperately. In this case the number of lines was too large to fit into the spreadsheet.</p> 
  
  
  <h1>Data Files</h1>  
  
  <p>The distribution comes with two example data files, Zoo and Mushrooms, 
  both courtesy of the <a href="http://archive.ics.uci.edu/ml/">UCI Machine Learning Repository</a> .
  Thanks to: Asuncion, A. &amp; Newman, D.J. (2007).   UCI Machine Learning Repository 
  [http://www.ics.uci.edu/~mlearn/MLRepository.html]. 
  Irvine, CA: University of California, School of Information and Computer Science.</p>

  <p>The Zoo data file is by courtesy of Richard Forsyth.
  It lists 101 kinds of animals with 18 nominal attributes
  (including the names). The original data have been altered by replacing the two
  instances of "frog" by "frog1" and "frog2",
  respectively, and "girl" by "giri". All numerical
  boolean instances in the original have been replaced by
  "Yes" and "No".<br>
  The number of reduced rules with "Type" as the consequent attribute is 823.
  These fit into 9 top level equivalence classes. On a 2GHz machine the reduction
  took seconds to finish, but the time depends on the structure in the data, and is different for
  different attributes.</p>
  
  <p>The Mushroom data file is by courtesy of Jeff Schlimmer, who lists its origin as
  drawn from The Audubon Society Field Guide to North American Mushrooms (1981).
  G. H. Lincoff (Pres.), New York: Alfred A. Knopf<br>
  The example consists of 8124 cases and 23 attributes, including "Class", which has values
  "poisonous" and "edible".<br>
  The original file is coded by letters, according to a supplied legend, but this coding has
  been reverted to the original names for readability. The missing value mark "?" has been replaced
  by "missing".<br>
  The number of reduced rules for "Class" is 3635, and there are 165 dependency trees and 5 Unconnected rules.
  The number of all implications and equivalences was more than 65535, too many for OO Calc to load.<br>
  Parsing the source data  and checking for duplicates (none present) took half a minute, checking
  for ambiguities (none present) also took half a minute. The subsequent reduction (consequent "Class")
  took 9 minutes and getting the top level then took a few seconds. Getting all dependencies, 
  resulting in a 150 MB large file, took 4 minutes more.
</p>

<h1>Notes</h1>
  <p>The Emping utility is written in Haskell, and has been
  developed and tested on the Fedora Core Linux platform, using
  the Haskell tools and libraries which are available as FC packages.
  Version 0.4 has been developed on FC8 with GHC 6.8.2 and Gtk2Hs 0.9.12.
  Earlier versions of GHC will probably not compile, and Gtk2Hs versions earlier than 0.9.12 
  don't support the implemented menu fields, and will not work.<br>
  <p>But GHC and Gtk2Hs are implemented on many platforms, including Windows and Linux versions, so
  Emping-0.4  should also work on those platforms.</p>

  <p>Any comments, bug reports, feature requests or remarks will be
  most welcome.</p>

  <p><i>Emping</i> stands for <i>empirical reasoning</i> or the
  Indonesian snack with that name.</p>

  
  <h1>Warning</h1>
  <p>For large data files like the Mushroom collection, which take a noticeable time to process, windows will
  remain open and the value selection popup may hang. This will correct itself when
  the processing is finished. Do not abort until you are sure something is wrong.</p>
  
</body>
</html>